Wget recursive download options
=====

``` bash
wget --recursive -np -nc -nH --cut-dirs=4 --random-wait --wait 1 -e robots=off https://site.example/aaa/bbb/ccc/ddd/
```

This downloads the files to whatever directory you ran the command in. To use Wget to recursively download using FTP, simply change `https://` to `ftp://` using the FTP directory.

``` bash
--recursive
```

download recursively (and place in recursive folders on your PC)

``` bash
--recursive --level=1
```

recurse but --level=1 don’t go below specified directory

``` bash
-Q 1g
```

total overall download --quota option, for example to stop downloading after 1 GB has been downloaded altogether

``` bash
-np
```

Never get parent directories (sometimes a site will link upwards)

``` bash
-nc
```

no clobber – don’t re-download files you already have

``` bash
-nd
```

no directory structure on download (put all files in one directory commanded by -P)

``` bash
-nH
```

don’t put vestigial site name directories on your PC

``` bash
-A
```

only accept files matching globbed pattern

``` bash
--cut-dirs=4
```

don’t put a vestigial hierarchy of directories above the desired directory on your PC. Set the number equal to the number of directories on server (here aaa/bbb/ccc/ddd is four)
	
``` bash
-e robots=off
```

Many sites will block robots from mindlessly consuming huge amounts of data. Here we override this setting telling Apache that we’re (somewhat) human.
	
``` bash
--random-wait
```

To avoid excessive download requests (that can get you auto-banned from downloading) we politely wait in-between file downloads
	
``` bash
--wait 1
```

making the random wait time average to about 1 second before starting to download the next file. This helps avoid anti-leeching measures.


## References:

+ [scivision](https://www.scivision.dev/wget-recursive-download/)
+ [wget recursive download](https://www.gnu.org/software/wget/manual/html_node/Recursive-Download.html#Recursive-Download)
+ [comparison table](https://curl.haxx.se/docs/comparison-table.html)
+ [recursive retrieval options](https://www.gnu.org/software/wget/manual/html_node/Recursive-Retrieval-Options.html)
